How Deep Learning Finally Cracked Messy Tables - Frank Hutter

Deep learning did not work for tabular data, and now it works dramatically better than CatBoost and XGBoost.

TabPFN is really the natural progression of AutoML, where we learn an entire algorithm that is executed in a forward pass. With 1 million rows and 1,000 columns, if you were to throw this into an LLM, you would have 1 billion elements. You need to tokenize each of these numbers, so that would maybe be 3–4 tokens each. You would have 3–4 billion tokens in your context, and LLMs would not be very happy there.

You’re getting some observational data, and you want to predict, “If I change this variable here, what will happen?” If I give this patient this medicine, what will happen?

Frank, it’s amazing to have you here. Welcome to MLST.

Thank you for having me. It’s great to be on the show. Two years ago, people asked me, “Hey, is there a possibility that you might be on MLST at some point?” I was like, “Maybe at some point.” It’s super exciting to be here now.

My name is Frank. I was a machine learning professor for the last 12 years before starting Prior Labs as CEO, and now I’m co-CEO of research. I’m very excited about that. I focus a lot on AutoML, or automated machine learning. I started the workshop series on AutoML and ran that for 8 years. We transitioned that to a conference on AutoML. I co-wrote the first book on AutoML and did the first MOOC on AutoML. I did a lot in that space, but then also moved over to deep learning.

I worked a lot on the optimization of deep learning, so AdamW and cosine learning-rate scheduling are from Ilya Loshchilov and myself. Those are being used for training essentially every transformer in the world. Then we put the two together and did deep learning for tabular data. That didn’t use to work, and with TabPFN, we actually made it work. We’re scaling this up, building foundation models, and are really excited to revolutionize this world.

I think one thing that people might not appreciate is that tabular data is absolutely everywhere. Speaking from bitter experience, it’s a nightmare to work with tabular data. Anyone who’s built a machine learning model with tabular data—you’ve got Pandas and Scikit-learn, which have some stuff in there, but you still have to deal with missing values. What do you do with the categorical features? How do you do feature engineering? How do you do transformations that reflect the semantics of the problem?

You really need to know what you’re doing, and it feels kludgy. Even though you can make an ML model, you always feel dirty afterward because you feel like you’ve taken all of these shortcuts and corrupted it in some way. Tell me about that.

Fully agreed. Tabular data is very dirty, and I think that’s one of the reasons deep learning took so long to actually do well on it. There have been countless attempts at deep learning for tabular data. In 2019, TabNet by Google was really hyped—thousands of citations—as the new thing for tabular data, but it just doesn’t work. It doesn’t generalize to new datasets.

You have heterogeneous data, outliers, missing values, categorical values, binary values, ordinal values, missing data, data missing at random, and data missing not at random. There are all kinds of data complexities, and you need to take that into account. The standard deep-learning approach treated everything as the same.

An image is the same in terms of having pixels, and you typically have very similar spatial relationships between the pixels. You can take images—for example, ImageNet, which covers all kinds of different images in the world—and learn on that one dataset. But what you need for tabular data is a whole lot of different tables. If you have one table from medicine and another table from insurance, there’s nothing you can learn in terms of the individual rows. One table can’t tell you anything about the other one.

What you can do, and where you can transfer knowledge, is at the level of these patterns: detecting patterns, how the different features interact, and so on. You can potentially detect causality. At that level, using in-context learning, you can actually transfer across different datasets. That’s why, with in-context learning, we finally saw this breakthrough for tabular data.

I’m having nightmares thinking back to doing stuff like this. Traditional machine-learning algorithms wanted the data in a Euclidean space within a certain range, so you might have to normalize or standardize it. Some of the algorithms, such as SVMs, might not work very well if the ranges of the different fields were different. It was an absolute nightmare.

Can you explain this a bit more? Before, the job of the data scientist might involve doing a whole bunch of cross-scatter plots, feature analysis, and so on. You would convert the data into this beautiful, convex Euclidean space. Now we have this system using a transformer, and I guess the architecture vaguely resembles Deep Sets, so it can actually intrinsically understand discrete data. This is very new. Why are we in a different world now from where we were before?

It can understand discrete data, and it also has architectures that work on individual columns, as well as different types of preprocessing in the network. The network looks at the statistics of a column and makes sure that you have the right embedding for that column. That’s all embedded now, so you don’t really need to do this yourself anymore.

Dealing with missing values, outliers, and so on used to involve a lot of preprocessing steps. Not having to deal with that is, of course, very nice and makes you so much faster. That’s one of the things we wanted to achieve with tabular foundation models: helping data scientists do their work so much faster.

I think that, in this day and age, new data scientists really don’t need to understand everything about XGBoost anymore—its various hyperparameters and what changes when you change them. The job really changes. It’s all about thinking about the right framing of the problem. Where does the data come from? Is there going to be data shift? Is there data drift? Are there problems with fairness, causality, or privacy?

When I deploy this model, what’s going to happen? How is this going to affect how the data is being collected? That’s a completely different ballgame, and there are very exciting questions there that people currently have no time to consider. Being able to spend more time—more human-quality time—on a problem rather than fiddling with hyperparameters is going to make being a data scientist a much more fulfilling job in the future.

I suppose another reason tabular data hasn’t been very sexy is that a lot of data scientists love what the deep-learning revolution brought: this notion of scale. You can go out on the internet. Even if I’m working in a bank and doing document recognition, my headspace is very much, “Let’s gather up all of the documents. Let’s see if there are other documents on the internet. Let’s build a dataset.”

There’s this notion that I can just scale, add more data, and it’s going to work better. Whereas the mindset with tabular data is very much a lack of imagination. People are thinking only in terms of the particular dataset they have, rather than how they can enrich it with other datasets or use a tabular foundation model.

Absolutely. Humans like to put their text online.

They like to put their images online and their videos online. It's very easy to make a viral tweet with a generative video model, et cetera. But people do not put their tabular datasets online, right? This is just not very glamorous, and it's also kind of abstract sometimes to think about tabular data.

But when it comes to our health, our science, and our financial system, it becomes very concrete, and it's very clear that this is a really important problem that needs to be solved. Having these tabular foundation models that are pretrained on synthetic data was the lever that actually allowed us to capture this.

This was necessary. We couldn't just take a corpus of tabular data that someone had prepared for us. We needed to generate our synthetic data because there is no tabular data—there is some tabular data, but maybe 50,000 datasets that are there. On the web, there are millions of datasets, millions of tables, but they are tables like those on Wikipedia: “This basketball player has this number on their back.” It's not a dataset that you can reasonably learn a statistical learning algorithm from. Those types of datasets just haven't been there, and we needed to generate them in order to learn from them.

And just to hammer that home, there has been an ImageNet moment for tabular data, right? Until the day before yesterday, figuratively speaking, deep learning did not work for tabular data, and now it works dramatically better than CatBoost and XGBoost.

TabPFN is the first algorithm that's actually been learned from data to be better at what it's supposed to do. We've pretrained it on hundreds of millions of datasets to be as good as it can possibly be on the test set, and that is not peeking at the test set. That's not what a lot of people do when they're overfitting or something.

You don't get to look at the test set at test time, but at meta-training time, over hundreds of millions of datasets, you have learned to really solve this problem. We're feeding the entire dataset in context to the transformer, and the transformer can attend to the right parts of the dataset in order to figure out what the patterns are in this dataset and which patterns it should then also apply to the test data in order to predict well.

And an important detail is that these guys have synthetically generated a whole bunch of the training data, and this overcomes the overfitting problem to some extent, maybe not entirely. You can explain that. But you synthesize data, and then you build this foundation model.

Absolutely. There are a lot of strong theoretical foundations behind this. We're approximating the Bayesian posterior over any type of prior that we can sample from, and the prior that we chose has a lot of causality in it. We're basically building the Bayesian posterior of all kinds of structural causal models that could explain the data.

In contrast to LLMs and vision models, we have trained on synthetic data. We had to train on synthetic data because there isn't a whole lot of tabular data on the internet. Rather, we're the first foundation model that's actually state-of-the-art yet entirely trained on synthetic data.

This is great because we don't have any leakage or memorization issues, et cetera. We can directly control exactly what's in the data. We don't have to worry about biases because we have a codified pipeline for generating the synthetic data from scratch.

You guys run this website. Many folks at home have seen LM Arena. That's using an algorithm related to the Elo algorithm in chess. In chess, you play against other people, and depending on the information gain from the result, you get a rank for every single player. It kind of converges over time, depending on the parameters.

LM Arena did that with language generation, and humans could rate whether one result was better than another. You've got this thing called TabArena. You actually run the site yourself.

TabArena started as a collaboration between a whole lot of different people in the tabular community. The first author is Nick Erickson, who at the time was at AWS. He built AutoGluon there, which was the best AutoML system for tabular data there was until now, when you combine it with tabular foundation models and it gets much better.

There were also a bunch of other people with, I think, 5 or 6 different affiliations. By now, Nick has joined our team, and several other authors have also joined our team because they all want to push tabular predictions to the max.

TabArena is very open. We include any type of new model. It's a living benchmark, so when we see issues in any of the datasets, or people say, “Why is this dataset not in there?” we update it. Or if someone says, “This is a new dataset. Let's update this in order to get less overfitting of the community to this dataset,” then we update it.

If you would like to get involved, by all means. We're continuously building new benchmarks. We have a benchmark that's now Beyond TabArena, which goes beyond TabArena. We're building a benchmark for relational data and for feature selection. All of these efforts are entirely open source, and we invite collaborations with anyone.

Is it conceptually similar to LM Arena? There are a whole bunch of different categories, and you sample a comparison, then a diverse population of humans rates one as being better. Is it a similar kind of thing?

In TabArena, you don't need to have that step where humans rate the results. You just have a training set and a test set. What is similar is that you compute Elo scores. You have these tournaments, and you can say, “Look, if the Elo score is 100 points higher, then the probability that this algorithm wins against another algorithm on one of the dataset samples is this high.” That is one similarity to LM Arena.

But it's basically an entirely open platform that publishes all the artifacts and all of the per-split predictions of all the algorithms being compared. It has been reproduced by people in the open-source community. That makes it much easier for anyone to benchmark the algorithms properly, and running your own benchmark costs something like $2 on a GPU. This is totally accessible to anyone.

That makes a lot of sense because with language models, it's not objective. Humans need to compare them because we can't design an objective function. That's a problem as well, because humans love it when Star Trek is mentioned, and they love GPT slop, and all sorts of weird things are going on there.

In this case, you have an objective criterion and you can run the thing. Just one final thing, though: the great thing about TabPFN is that you can synthesize data, and what we don't want in benchmarks—I mean, it happens grotesquely with normal benchmarks, because sometimes you're just training on the benchmark—is there an inadvertent form of leakage here where you're continuing to generate more data and you can kind of see what's going on in the benchmark? Is there a tendency to inadvertently leak some data?

It is not the case that we synthesize data and then that data goes onto the benchmarks. The benchmarks are all real datasets that come from the community. We curate them from thousands of different possible datasets, then we drop them for various reasons, and every step is exactly documented.

For example, for Beyond TabArena, we dropped everything with fewer than 100 samples. We could have included those, but with fewer than 100 samples, maybe you don't have as much signal anymore, et cetera. We also drop duplicates.

For example, you could see MNIST as a tabular dataset by taking every pixel as a number. But we drop those datasets because we only want datasets where you would actually use a tabular machine-learning algorithm and where that would make sense, in order not to get distracted by datasets that look tabular but where you should really use a vision classifier, as with MNIST.

There are spatial patterns and spatial similarity between neighboring pixels that you want to exploit. If you cast this as a tabular dataset, you sort of pretend that you have that type of data in tabular data, and you typically don't want to have those patterns in tabular data.

We drop datasets for various reasons that are very well described. Again, all of these papers are open-source collaborations, so we're happy about really anyone who wants to put in the effort to curate these datasets. This is terrible work, much easier now with agents.

Still, it's hard to define correctly what the inclusion characteristics should be. We run these experiments, keep the website up to date, and look at all kinds of Pareto curves: how good is the predictive performance, and how good—or how fast—is it in terms of training, inference latency, et cetera? We also look at all kinds of statistics of the competing methods.

And apparently, you did your PhD under the legendary Kevin Murphy.

Yeah, indeed. Actually, I had 3 supervisors: Holger Hoos, Kevin Leyton-Brown, and Kevin Murphy. Kevin Murphy was maybe the least believer in automated algorithm configuration, but he was still excited about some of the things we could do. In the end, when he went to Google, he became excited about neural architecture search.

Yeah.

So then we actually collaborated more than before.

You've got a really interesting background because I remember when I was a data scientist, about 10 years ago, there were lots of discussions about things like AutoML and neural architecture search. In fact, you were one of the people behind that NAS paper. I remember reading about that at the time, years and years ago. Sketch out that history.

Yeah, AutoML goes really way back to my PhD. My PhD was in automated algorithm configuration, which was all about making all of the nitty-gritty decisions in designing algorithms more automated. It came from SAT solvers, where you needed to make hundreds of different decisions in categorical spaces, and this was really annoying.

In my master's, I wrote a local search algorithm for one particular problem and worked on different types of problems. You always needed to make all these decisions, so I started automating that with local search to find better parameters for my own local search algorithm. Then I got exposed much more to machine learning and used machine learning to find better parameters for my SAT algorithms. After that, I moved more toward using machine learning to find better hyperparameters for machine learning algorithms.

That led to model selection, choosing one of different types of algorithms. Each different algorithm would have a lot of hyperparameters of its own, and then you have these hierarchical, really complex spaces. You want to do optimization in that space efficiently, but you also want to generalize to different parts of the distribution. You don't want to be stuck and over-tuned to one particular type of problem distribution, but really generalize to all kinds of problems you're going to see afterward.

Then there was this other design space of neural architectures, to which the same types of approaches really applied directly. It's very categorical and very structured, with all kinds of hierarchical decisions. We pioneered Bayesian optimization for that. We also definitely saw a lot of issues in neural architecture search in terms of how to benchmark different algorithms and so on. It's very complex and costly to run these algorithms, which makes it very hard to do good empirical science.

Exactly. I remember when I was doing my PhD all those years ago, I was using something called Weka. You could try, “Let's use a support vector machine. Let's use a Bayesian network. Let's use kernel ridge regression.” It was a wonderful toolbox where you could say, “I've got a prediction problem,” and you had a bunch of data, signals, and labels. You could prototype all these different approaches and do cross-validation.

In the olden days, machine learning was mostly manual, and there was some hyperparameter optimization and so on. But I think this AutoML thing was hinting at this structural component. Wouldn't it be better if we could search the space of machine learning model structures with respect to the particular problem?

Yeah, absolutely. I love that you mentioned Weka because that was the first base library of classifiers that we were working with. We did this Auto-WEKA. Remember Weka? The image for this was a Weka bird, so we had a little robot riding a Weka bird to be Auto-WEKA.

Yeah, you described it. All kinds of people use this, often people from the sciences, life sciences, and biology who are not machine learning experts. There's all this stuff in Weka: dozens of different classifiers, dozens of preprocessors, and all kinds of different hyperparameters for them. What would people use? I don't know—an SVM with default hyperparameters because their friend told them that would be good. That's just not how you get the best performance.

You really want to automate this in order to have better approaches for the masses. That was the leading principle behind AutoML: to democratize state-of-the-art machine learning for everyone, including those without a PhD in machine learning.

I suppose the problem with Weka was that I was always so excited about neural networks, but in Weka, the neural network model was incredibly slow, and it was always worse than using simpler models. I suppose then we had frameworks like Keras, for example, and we started experimenting with different types of activation functions. Maybe we could have a CNN layer here and an MLP there, and it's very composable.

You could almost just construct a neural network architecture and compose together frozen ones that you'd built previously, or fine-tune them. But again, we're back to this incredibly manual process. How did automated ML touch the neural network space?

Basically, as you describe, people were annoyed by having to do this manually and were thinking, “How can we actually automate this space? What should the design space look like? What are typically the right elements in the space? How wide should I make my network, and how deep should I make my network?” Those were the early design spaces.

Then convolutional neural networks came around, attention came around, and you had all kinds of different hybrid architectures. On top of that, for different chips, you had different latency and so on, so you had hardware awareness in there. Then you had the performance of the algorithm, such as accuracy, but you also had latency and memory consumption.

You had all these different objectives, so it became multi-objective. It was just a big playground for very exciting method development in order to search through these spaces.

Amazing. I suppose there's that old story. As you were saying, in your PhD, you were specializing in SAT solvers, which is just: how can I search the space of possibilities? Something really big happened, I guess it was around the GPT model moment, where we had this concept of a foundation model.

Rather than data scientists trying to build low-level models and search the space of possibilities, what they increasingly were doing was using these foundation models that had been trained on loads and loads of data and working from that as a starting point. What was that transition like?

There was a really exciting transition. ChatGPT came out in November 2022, and a week before that, we published the TabPFN-1 paper, which was the first foundation model for tabular data. It completely turned tabular data on its head. You didn't need to do this model selection anymore; you would have one pretrained model that would use the entire dataset in context and learn across millions of different datasets how to make predictions for the test data in one forward pass.

One particularly exciting part of AutoML over 15 years was the story of meta-learning: learning the parameters of neural networks across different types of datasets in order to work well on unseen datasets. You can think bigger and think about learning entire algorithms that generalize to new types of datasets.

TabPFN is really this natural progression of AutoML, where we learn an entire algorithm that is executed in a forward pass. Depending on the dataset that you feed in, it learns a different classifier. It's not just one network that is one classifier, but a network that can learn different classifiers depending on the input.

You could take this network, write it out as ONNX, and put it on a sensor. Then you have a sensor that runs not just a classifier, but a machine learning algorithm that, depending on the inputs, will actually give you different classifiers as output.

And so you can see this as, for example, traditional algorithms like XGBoost. All of these algorithms are hand-derived, hand-coded, and so on. TabPFN is the first algorithm for tabular data that's actually fully learned end to end over a large set of different data sets that it's supposed to work well on.

We're trying to optimize a cross-entropy loss on the unseen test portion of each of these data sets. When you have a new data set, you of course don't see the test set. You can't peek at that. But at training time, that is the loss metric you actually want to optimize: you want to do well on this test portion.

We look at hundreds of millions of data sets and make sure that the algorithm works well on those, and then it will also generalize to new data sets. That is the beauty of AutoML for algorithm development. It lets you be really declarative. You can say, "For these types of data sets, you should work well."

For TabPFN 1, we had a fairly simple set of data sets. Then, over the years, we made this more and more complex. For example, with TabPFN 2, we put in missing values and outliers, and all kinds of other data complexities like that. Uninformative features were something that was broken before. Categorical features were something that didn't work well before. We put more and more of these complexities into the creation process for the data sets that we would train on.

The objective was to do well on data sets with these types of data complexities. Neural networks and deep learning just need to do their magic and actually optimize for that objective, and they can do that well by pure supervised learning. There's no magic there. Then out comes an algorithm that executes in a forward pass, and we don't have this long training loop anymore. We don't have the search over hyperparameters at test time, but we can just do a forward pass.

All of the other parts of AutoML are still there, but they're there in coming up with this algorithm. We can still do a search over the hyperparameters of the architecture of our neural network. We can search over our learning rates for the pretraining, and so on. But all of that falls away at test time, and data scientists get what they always wanted: a really fast method that works out of the box.

Now, I've just got this model, and the next component is what you were talking about, this adaptive inference. So it's very, very similar to chain-of-thought adaptation in a normal transformer. You put a prompt in, and what it's actually doing is conditioning the transformer. It's doing some kind of computation that's specific to the input that you give it. So in your context, without actually training the thing from scratch, you can put some tabular data in there, and you're saying that, in a single forward pass, it is essentially creating a model that works specifically for your case.

Yeah, absolutely. This is literally a single forward pass. With chain-of-thought and LLMs, you're doing this autoregressively, so you produce one token at a time. But with tabular foundation models, it's literally the one token you need to predict, and that is it.

You have your training data, X train and Y train, the label, and then you have X test, which is the data point that you want to predict for, and you only want to predict Y test. If you have many X tests, then X test 2 shouldn't depend on what you predicted for X test 1. So there's this independence, and it's really just one token at a time. You don't roll out autoregressively.

That lets us directly optimize for the objective function that matters in the end.

So you've spoken about how this is conceptually similar to Gaussian processes, and I suppose we should also contextualize that. In most machine learning models, even in current transformers, you can say that they're approximating some kind of Bayesian model, but they're approximating a maximum-likelihood point estimation. Whereas your model is actually approximating the full Gaussian process. It's actually approximating the uncertainty across all of the possible values. Why is it possible for your model to do that and other models don't? What's going on there?

What we approximate is directly the Bayesian posterior predictive distribution. What other types of Bayesian inference methods typically do first is get a posterior distribution over the functions.

Maybe to back up a little bit: you have a prior over functions, or equivalently, a prior over parameters. Then you observe some data, and you want to reason about the posterior over a function or the posterior over parameters that describe the function. For computing that posterior, you can use Markov chain Monte Carlo, or you can use variational inference, and both of them have their issues.

MCMC is just really slow, and variational inference involves complex math. The approximations sometimes don't work out perfectly, and it's also sometimes a bit slow. Once you have this posterior over functions or parameters, then typically, in order to get the posterior predictive distribution—the P of Y given X and the data—that is actually typically easy.

If you have a sample-based approximation of the posterior over functions, then that's basically just a sum over that sample-based approximation. The tricky part is that this is a posterior over the latents, over the parameters or the functions. We just entirely skip that step. We go directly to the Bayesian posterior predictive distribution, which is a one-dimensional distribution. It's just the P of Y. Y is a scalar for regression, or, for classification, it's the probability over the K classes. We can do that in a forward pass.

How do we do that in a forward pass? We sample from the prior. We sample functions from this prior over functions, and then we sample data points from each of these functions. If you think of Gaussian processes, we have a Gaussian process prior, and we can sample different functions from that. You have some kernel specifying the Gaussian process, so that's telling you something about how bumpy it is, and so on.

You sample functions from that, and then you sample data points from each of these functions and call some of them your training points and some of them your test points. You basically learn to predict the test points, or the Y value of the test points, from the training points.

If you can do that for millions of samples of this function, then you have actually learned to capture the structure of the prior in your network. The network has learned to approximate this posterior distribution over what the Y value should be at these missing test data points, and has learned that over these millions of samples from your prior.

Given a new data set, this is the first time you actually have a data set. Before that, it was really just samples from the prior. Then you have a real data set, and you can compute the Bayesian posterior predictive distribution for that in a forward pass, just like you computed it for each of your millions of training data points.

You basically meta-learned this posterior predictive distribution, or meta-learned to approximate the posterior predictive distribution for arbitrary data set inputs, purely through 2 things. You need to be able to sample from your prior, and you need to be able to fit strong neural networks in a supervised learning fashion.

Both of these are actually quite easy if you have a mechanistic prior that you can sample from. That's been our workhorse for improving our priors. Then, for the deep learning machinery, we just piggyback on what folks do in LLMs and transformers. Architectures get better, optimizers get better, and our methods also directly get better. Of course, we adapt the architectures to tabular data as well.

Yeah. We can compute this posterior distribution. Traditionally, in Bayesian models, we integrate over all of the possible hypotheses, and it's possible in this space—because I want to try to distinguish how this is different from something like transformers.

What you've done here is train these models using synthetic data. You've come up with a whole bunch of priors about causal relationships in different types of structured data. You couldn't do this with language, for example. You could have a context-free grammar, and you could generate a bunch of language, but it would just be some weird, inscrutable language that we'd invented.

The really cool thing here is that you can say, "Okay, there are these principles that are common to tabular data, and I can represent those as priors. I can sample from them." In your predictive architecture as well, the output space might be discrete. It might be categorical, which means there are relatively few examples. Or, if it's a regression problem, you can discretize it to a tractable number of things.

So essentially, you can do this thing end to end in a way that wouldn't be possible with many other types of models.

Yeah.

That's absolutely true. If your prior is, for example, just—say it's really simple; there are only linear curves—you can sample from that. What will come out as a Bayesian posterior predictive distribution is actually Bayesian linear regression. If you sample from Gaussian process priors, what comes out is a Bayesian posterior predictive distribution, which is actually just the GP posterior, or an approximation thereof.

If you have a Bayesian neural network, or a neural network that you sample from, then out comes a Bayesian neural network prediction. Again, not a posterior over the neural network parameters itself, but a posterior predictive distribution. Take all the possible neural networks out there from your Bayesian neural network and integrate over them.

If our prior is the space of structural causal models, then our posterior is the integral over all possible structural causal models that could cause the data. For each of them, how likely is this SCM to cause this particular data? And what would the SCM actually predict for the test points? Then you integrate over all of them.

Of course, there is a huge number of them. It's probably countably infinite. But we only see a limited number of them, like hundreds of millions. Our network is so small that it can't possibly memorize that, so it actually learns to generalize across the space, even just from seeing a finite number.

So I suppose the interesting thing is that you can scale this up and, at some scale, it will asymptotically converge on what the Gaussian process would have been. It's already reasonably close to that, based on the graphs that you've shown. This is incredibly exciting because what we haven't said yet is that you can think of this like the ImageNet moment for tabular data.

When you look at TabArena—was it around 25 or 26?—there was a step change. Focus on Kaggle: they're using XGBoost and CatBoost and—what was the other one?—LightGBM, is that right? Yeah. There was this absolutely massive step change.

A couple of things are going on. Now you can generate essentially as much training data as you want, and you can bake it into the model. I think the important concept is that it's doing a kind of amortized inference. Whereas before, when you were doing training and inference, you had to do a lot of work. Now you have a single forward pass, which means essentially you have a fixed amount of computation.

Exactly. We basically learn this network that's trained in order to do as well as possible over the types of datasets that we feed in as input. Historically, what we've done is generate relatively simple datasets with TabPFN 1—super simple—and TabPFN 2, which was a bit more complex, with outliers, et cetera. But even with TabPFN 3, they're still relatively simple IID datasets.

There is a relatively low number of data points compared to the billions of data points that are out in some datasets. With TabPFN 1, there were tens—what? One thousand data points. TabPFN 2 had 10,000 data points. TabPFN 2.5 had 100,000 data points. TabPFN 3 has 1 million data points.

We're getting there—about 2 orders of magnitude a year. But we're still not at 1 billion, and a lot of the datasets on Kaggle, for example, are so large that the tabular foundation models haven't caught up there when you use them in a forward pass.

You can have a harness around it, where we have scaling mode and thinking mode, et cetera, and you can add wrappers around this and use test-time compute in order to scale up. Then it could also work for the data complexities of Kaggle. But for now, it hasn't been used for that many Kaggle competitions because there aren't a whole lot of Kaggle competitions in this really small-data regime of around 10,000 data points that we could deal with using TabPFN 2.2. We can tackle 100,000 since last November, and we can tackle 1 million since about 2 months ago.

There just haven't been a whole lot of Kaggle competitions of that size. In general, there just aren't that many tabular competitions anyway, because language and vision, et cetera, have gotten a lot more attention. But tabular data is so important in the world that I definitely see a lot more focus on this type of modality now again, and that will also lead to more Kaggle competitions in the future.

We're focusing on ever more complex datasets in our prior generation—for example, non-IID group data, temporal data, text in our tables, et cetera. All of these are data complexities that you see everywhere on Kaggle and that were just nowhere in our benchmarks, nowhere in our data-generation processes. We're changing that.

It's in our benchmarks now. We have this Beyond TabArena that has all these data complexities in there. That shows that TabPFN 2.6 was not particularly good for some data complexities. TabPFN 3 helped on some of them, but it's still not quite there. That's a signal we're optimizing for right now.

Yes. I think what—let's come back to the multimodality stuff later, because that's really interesting. A good mental frame for this is that, when you use a normal language model, there's a context, and you have this quadratic complexity, so things just slow down. If you're using Claude, for example, when you go up to 1 million, it's really struggling, and it's slowing down a little bit.

I think it's a similar thing here: the context is essentially the table that you put in. Is it around 100,000 rows now?

Yeah, so it's up to 1 million rows.

Or 1 million rows, yeah.

Up to around 1 million rows times 1,000 columns. If you were to throw this into an LLM, you would have 1 billion elements, and you would need to tokenize each of these numbers. That would maybe be 3 or 4 tokens each, so you would have 3 or 4 billion tokens in your context, and LLMs wouldn't be very happy there.

They just don't work out of the box for this type of data. They're not made for tabular data. They would just need to read the table in some sort of sequence because they're sequence models. If they read 1 row at a time, then they don't understand that if you switch 2 rows, or switch 2 columns, it's the same dataset.

That's why they don't exploit these invariances. They have a whole lot of parameters that are super important in order to capture the semantics of the world and world knowledge, but that you don't actually need in order to capture the statistical reasoning that you need to serve this standard interface of XGBoost, et cetera, where you just have X_train, y_train, X_test, and want to predict y_test.

There, you don't even see column names, so you really don't need to know anything about semantics for this. If you do know semantics, then you can do more. If you have 5 rows or something like that, and you know, for example, that this is a churn dataset and you know exactly what the column names are, then world knowledge comes in, and LLMs would be great.

But if you have 1 million rows, you actually want to learn about the statistics of these numbers, and that's where LLMs really fall flat on their nose.

Yes. I suppose we should contrast what the trade-off is here. I could train an XGBoost model, and it would be very expensive. But I suppose the good thing is that I can then do streaming, parallelizable, row-wise prediction, and I can go through the dataset.

The good thing about your system is that it's actually taking the entire table—the entire dataset—as an input, which means it's potentially learning global relationships, first- and second-order relationships, and table-wise invariances. It's modeling at a level of sophistication that is far away from XGBoost. The beauty of it is that, because it's a foundation model, it just does it automatically.

It has learned to do this in a forward pass. Since you mentioned that XGBoost can stream a lot of predictions in parallel, historically TabPFN was very slow at inference time because you always had this entire dataset that you passed into the forward pass. There wasn't really a difference between the training stage and the prediction stage.

Now, just like in LLMs, we have a KV cache, so there is a difference between training and test. At test time, all you need to attend to is this KV cache, and you don't need to attend to each of the training data points.

You also don't need to keep your data around because, also for privacy reasons, you don't want to feed all your data when you make a prediction. You just need to keep the weights that are stored in your KV cache. It's much faster now, and on GPU, it's getting close to XGBoost prediction times as well.

We were contrasting this to something like AutoML before.

But you had quite a good slide, actually. I was reading your deck before I came here, and you correctly observed that if you fed, let’s say, 1,000 rows into a language model—a bare language model—it’s not going to do anything interesting, is it? It doesn’t understand the structure in the data. It’s not modeling those invariances.

But I’m just not sure that’s an entirely fair comparison, because now we live in the world of Claude Code and Codex. I could say to the agent, “Here’s a load of data,” and it would generate Python code. It would load it into a data table, and it would understand a lot of the semantics. It would understand what this meant, or that this is a risk thing, an insurance thing, or an accountancy thing.

It would be building all of these structured models, and I could tell it to crystallize a machine learning model. I could say, “Okay, well, there’s some text data here. Let’s use the MiniLM encoder and have an XGBoost head on it, or whatever.” The beauty of it is that it’s adaptive.

So next week, when there’s a prediction error, Claude will understand why the error happened, generate some synthetic data, put in some symbolic rules, and say, “Well, if the recall is less than this, then I’m going to escalate it to a Claude Haiku model,” or something like that. You see, now we’re in this world—it’s almost like the dream. A few years ago, wouldn’t it be amazing if we could have this? So how does it compare to some kind of adaptive architecture like that?

Yeah, I’m absolutely in agreement that LLMs are amazing for coding. Agents are great for feature engineering, exploratory data analysis, and as a user interface. They can actually let you double-check that all of the data is entered correctly and chat with you about where the data actually comes from.

For feature engineering, they’re amazing. I mentioned that TabPFN would be published about a week before ChatGPT, and when ChatGPT came out, we were so excited. The first thing we did was drop TabPFN in and actually do agentic data science with ChatGPT to do feature engineering on top of TabPFN.

You do need world knowledge for that feature engineering. A super-simple example is if you know the height and weight of a patient and want to predict something for which it would be helpful to know whether they’re obese, then of course you can compute body mass index. It’s just a simple computation.

You could do this computation, or you could have the network learn to do these types of computations early on, in the 1st layer, and so on. But if you can just do this outside with a coding agent, that’s so much easier. That was the first thing we did: write this CAFE paper that did automated data science with LLMs.

That’s the type of relationship I absolutely see. For data preparation, cleaning, and so on, LLMs are fantastic, and we’re using them left and right. But in the end, they need to call a model. They can call XGBoost, but it’s just not going to be as good as the tabular foundation models that are getting better by the day.

I suppose we could have our cake and eat it, because what people can do now at home is tell Claude or Codex to go and grab TabPFN v3. For noncommercial use, it’s free. They could actually get the agent to use TabPFN.

Here it gets a little bit interesting, right? I think one of the great things about TabPFN is that our datasets and tables are legible. But what if Claude did weird stuff? Sometimes, to get better representational friction, you do weird, inscrutable feature engineering and whatnot. Maybe Claude does that. Maybe Claude takes 2 of our features and turns them into 100 features, and we end up with a spaghetti mess. Would that be a problem?

I don’t think it would be too much of a problem, because you would prompt Claude to tell you exactly what it’s doing and give you the code and the features. That’s actually quite similar to what a data scientist would do, right? They write some sort of feature-engineering pipeline, and that might get a little messy over time. But it’s simple code in the end of the day. It’s simple operations that you apply to your different features.

What Claude would also do, if it’s doing a good job, is access all kinds of feature stores online. If, for example, you have a dataset and one of the columns tells you about the country that the data point is from, and maybe it’s a shopping dataset, then you can say, “Well, in this country, this particular day is a holiday, so there’s going to be more demand because people have more time to go shopping online.”

That’s going to be a super-important feature that you can then generate: Is this a holiday in this country or not? That requires external information. You can’t just see this from the context, and Claude probably doesn’t want to just remember this, but actually look it up in some knowledge base. Of course, it can do that.

You can now use Claude with TabPFN, as we have an MCP server that’s integrated with Claude, integrated with Gemini, and also with all kinds of other frameworks. It’s really easy to get going. You can just say, “Hey, Claude, build me a dataset for this problem,” and then use TabPFN to predict.

A friend of mine used this for the World Cup. We had a little bit of fun and made predictions for every game. That friend actually used Claude to prompt it: “Hey, Claude, please build me a dataset from historical games, or any way you want—a tabular dataset—and then use TabPFN to predict every game.” With that, they actually won their cake bet at their company.

You can now have your cake and eat it. Absolutely. To me, this is really the convergence point of AutoML, because we can give state-of-the-art machine learning to novices who don’t even need to know how to code. They don’t really need to know what a tabular prediction problem is, and the friction is just so low. At the same time, you still get state-of-the-art performance.

We actually have a variety of different demos. We have 1 demo integrated into Databricks, where you can look through your dashboards and ask the agents questions. For some of the questions, you actually want to do a tabular prediction, and it will just do that. It won’t even tell the user that it’s using TabPFN. Why should it? It’s just making better predictions.

That’s definitely the world we’ll be in within a couple of years, where you’ll see tabular foundation models used left and right, often without you knowing it.

Yeah, I’m genuinely excited to try it myself, because it’s one of those things that has utility almost everywhere, especially if you can get a coding agent to help you do it. That’s amazing.

There are some other advantages that we haven’t spoken about. For example, you can use them to generate data, do things like density estimation, or even generate embeddings. I guess we want to get away from the idea that it’s only used for prediction. You can actually use it for explainability or as part of building some other predictive architecture.

Yeah, and I think that’s super important, right? We need to move beyond just classification and regression. It is a foundation model; it can do all kinds of things.

We want to push towards relational data. A whole lot of data that is tabular actually comes from merging 2 different tables in a relational database. Working directly on the relational data would be so much more powerful.

There’s time series, which is actually very related. There’s causality, too. We haven’t talked about that much, but it’s really core, because you do want to make distinctions between causation and correlation. It’s so helpful to be able to understand the world in a causal manner; then you understand it much more deeply.

This is a big challenge, and causal machine learning is a big field. I think we can make a very solid dent in it with tabular foundation models.

Yes, and I think we should come back to that, because that’s very important. Right now, people can just import this library in Python, and it’s very similar to scikit-learn or something like that. You can just say, “I’ve got some data here. I’ve got my signals and my labels,” and, out of the box, it does regression and classification.

Now, if I understand correctly, you can’t combine classification and regression because you’ve decided architecturally to specialize in those 2 models.

So if you want to combine those modalities, I think you need to create 2 models. Is that correct?

It helps to have separate models for classification and regression sometimes because the gradients propagate a bit differently, and it’s a bit tricky to train both at the same time. But if you can train both at the same time, that’s of course so much nicer. We’re also working on that. We also want to have a causal head and a time-series head, et cetera, and just train all of that jointly. That would be so much nicer.

Can you explain that? When doing regression, you’re using an MLP head, and when you’re doing classification, you’ve got what I guess resembles an attention head, so you’ve got a special kind of decoder head for that. Tell me about the rationale there.

It’s not a problem to have different types of heads for different tasks, right? For example, for causality, we would need a different type of head than for the other methods. The way we used to do regression was regression as classification, where you just have this binning distribution, where you bin the space into 10,000 classes. Where a lot of the data falls, you have very small bins, and where not so much of the data falls, you have large bins, so that there’s the same number of samples in each bin. Then you just train a standard classification model.

The rationale for that was that, back then, my PhD student, Samuel Müller, did these experiments. He also tried a regression head that just predicted the mean and the variance, and it was simply better to predict in this classification manner. One of the reasons for this might have been that transformers, with all the hyperparameters, et cetera, are just made to do classification really well. Having 10,000 classes was not an issue at all, so we stuck with that.

One of the really nice side effects of that was that we could do these multimodal predictions. We could say, “Well, I don’t know whether, when the bird flies toward a pole, it’s going to turn left or turn right. It’s not going to hit the pole.” So you’re going to have a distribution here and a distribution here, with pretty much zero probability mass in between. You can do these predictions again in a forward pass, so you can have really well-calibrated outputs with this type of distribution.

Can you tell me about how you do the representation? There’s a bit of a lineage, right? Version 1 more resembled a normal transformer, just tokenizing things. If I understand correctly, now, in versions 2 and 3, it’s far more structured, and you have a specific way of mapping the attention because the transformer is permutation equivariant. Maybe we should start with that.

The architecture for TabPFN v1 was very much like a transformer, except that you drop the positional embedding because attention is already invariant to order. The only thing that makes a transformer pay attention to where the words are in the sequence is the positional embedding. We didn’t want that. We wanted to be invariant, so we just dropped the positional embedding. It was the simplest thing we could do.

What that required, though, was taking a row and encoding it into an embedding. We used a super-simple method for that. In TabPFN v1, we only had continuous features, and we just used a linear layer. That was it. That’s maybe not the best because, if we applied it to categorical values, we would just encode the categorical values as 1, 2, 3, 4 and then put a linear layer on there. Of course, that’s not the best.

In TabPFN v2, what we did was have an architecture that actually knows about rows and columns. We had an embedding for each individual element of the matrix, so for each value. Then you would have attention over the rows and attention over the columns, and you would alternate these. You could really understand, “Aha, in this column, the numbers are 1, 2, 3, 4. Probably this is categorical,” and treat this differently from a numerical 1, 2, 3, 4. We also had much more categorical information in the prior already in TabPFN v2, so there was better and better treatment of these categorical parameters.

Complexity-wise, I should talk about that because TabPFN v2 was actually much worse than TabPFN v1. TabPFN v1 was only quadratic in the number of rows, but it’s sort of bad because of the encoding of the row into a single embedding, and that is really inherently limiting. In TabPFN v2, since you had so many embeddings, for each embedding you would have this attention over all the rows and all the columns, because there are that many elements in the row. You have N rows and M columns, and then you have N²M + NM². That is the complexity of TabPFN v2, and that was okay for up to 10,000 data points.

If we wanted to scale higher—to 100,000, 1 million, et cetera—we needed to go beyond that. For 100,000, with TabPFN v2.5, we still did that. We got away with just moving to beefier GPUs. At the same time, the G-Research group actually developed TabICL, which was really nice. It was an architecture that basically, in the end, used the TabPFN v1 architecture, only quadratic in the number of rows. But it had a much more complex mechanism to actually do this embedding through a different column and row transformer.

Since that architecture was good enough for the size range we wanted for TabPFN v3, and also in terms of inference speed, we basically just adapted that for TabPFN v3. We innovated in many different ways—for example, for many classes and for the output head—but we didn’t actually have to change the base architecture. We have a lot of things cooking, but we didn’t need them for our desiderata for TabPFN v3, so they’re going to be in future versions.

Of course. But just to be clear, what is the complexity of v3?

Of v3, it’s actually the same as TabICL. Basically, it’s N² for the number of rows.

Yeah.

But then it’s NM² for the preprocessing, for the embedding—

Okay.

So it’s already faster, but it’s still quadratic, and there are a gazillion methods to make it subquadratic. Of course, we’re looking at those.

Yeah.

The next target is 10 million, and we’re pretty confident.

That’s super interesting. Even the rows and the columns—that is a wonderful example of the kind of inductive prior that we put into these models, right? We have certain assumptions about the data, and it seems reasonable that there are relationships row-wise and column-wise. But what would fast attention look like? For example, if you were going to cut down that span of attention, would it be some kind of locality prior?

In language, you have this locality prior. In tabular data, you don’t, right? Because you want to have this invariance over features. But you could have some sort of locality in the embedding space.

Very cool. I am very excited about test-time adaptation. o1 came out, and that was remarkable. The models became intelligent because, through this RL training, they were learning to prompt themselves. They were actually deliberating and thinking.

What a lot of folks have been doing with these language models is test-time adaptation, where they deliberately do some kind of structured inference at test time. It just feels like this is the perfect opportunity to do that. At the moment, you’ve made a deliberate decision to have an inductive model, and that has some interesting properties, certainly in terms of caching, calibration, performance, and that kind of thing.

If it were transductive, what that would mean is that the test samples could actually bind to the training samples, and that could be bad. Let’s say I’m doing a hospital prediction thing, and I might have some weird patient who’s an outlier. Now this other patient is affecting my prediction of the first patient. So it’s not necessarily a good thing, but it can be an incredibly powerful thing. Is that interesting to you?

I don’t think I would go too much in the transductive direction. It can go into the semi-supervised version, where you have a small number of labeled data points and then a whole lot of unlabeled data, and you know this is the type of data you actually want to work well for.

But in the end, I think I would stick with an interface where what I'm making predictions for doesn't affect the predictions for the other data points, because data scientists just wouldn't have that. I think that would be so weird: if you predict the same data point in a different batch, you get different results. It would be super strange.

Yeah, it's interesting. In the ARC Prize, for example, the reason it works so well is that you only have a few examples of something, and those other 2 examples give you so much information. We're desperately trying to adapt the model.

But I suppose another thing is adaptive thinking in general. I could imagine—and I think you've got something like this, although I don't think it's publicly known what you do—just thinking out loud here, that you might have some kind of latent variable, and you can have a latent representation that evolves with more computation. I do more forward passes, and I'm actually evolving some kind of representation.

Is that something you think about? Right now, we have a fixed amount of computation, but there are many tasks that we're uncertain about, where we might need to think, search around a little bit, and get a better prediction.

Yeah, there are a whole lot of analogies here. For some cases, you might want to make a prediction really quickly because it's a simple data point, and so you could have an early exit from a big network. For harder data points, you want to go all the way through. For even harder data points, you might want to think a lot harder.

There are a whole lot of ways of thinking harder. You could do fine-tuning: you could look at your dataset and say, “This is my dataset. Let's generate other datasets like this and do fine-tuning on that.” One of my master's students had a paper about that, showing that for very small datasets it can actually improve performance.

Then you can do prompt tuning. If you have a very large dataset, you could say, “Which parts of this large dataset should I actually feed into my context?” Say you have 1 billion data points. You can select 100,000 data points to put into your context, and you could do that in a gradient-based manner. But you could also hallucinate 100,000 data points that approximate those 1 billion data points better. We also had a paper at NeurIPS about that.

One fun fact there is that you can also use this for interpretability. One of my previous PhD students, Robin Schirrmeister, took a medical dataset of 1,000 data points and condensed that into 2 data points that you would feed as a prompt. That had the same predictive performance as feeding the 1,000 data points. That gave you the prototypes of the data.

There are a whole lot of things like that that you can do. You could also have different networks that were trained with different priors and do standard cross-validation on them. There is work that shows that transformers can actually do model selection in a forward pass. You could train a transformer with 10 different priors and have it choose the right one at test time in a forward pass. There are so many different things you can do, and I think the sky's the limit.

Yeah, exactly. There's also sampling and ensembles, and you could have some kind of scratchpad. I'm just really excited about this in general, though, because I wonder whether there's a tension there. We're couching this as a foundation model without adaptation, but it feels like adaptation is always strictly better.

This is why I'm quite excited about using agents, because new information comes in and we crystallize and optimize, and we end up getting something better. It feels like this could become the base of an architecture to do that.

The dataset distillation is really interesting. I think I interviewed a guy called Andrew Ilyas from MIT. He had a great paper on dataset distillation and machine teaching. The basic idea is that if you prune and distill the dataset, you can get dramatically better performance. This can be an online adaptive process because when folks are implementing this technology, they live where they live. They have very specialized data, things are changing, and this adaptation process could be part of the predictive architecture.

Yeah, absolutely. There are so many cool things, as I said, that you can do that you couldn't do with methods like XGBoost, for example. You could do feature importance in a forward pass, or you can do a forward-backward pass and say, “If I change this feature, how much would my output change?” You get a gradient signal for that, so forward-backward is trivial.

You could also do that for a data point. How important is this data point for training my model? That means you can actually detect adversarial attacks, et cetera. You could say, “This is really strange. This is one data point that completely dominates my model. How about I train again without that data point, and then see whether that is much better?” If so, then let's iterate this process.

You could figure out which features are important, train with only the important features, and drop the others. There are so many different things you can do there.

Let's talk about interpolation versus extrapolation. My good friend Randall Balestriero had this paper, “A Spline Theory of Deep Learning,” and he was talking a lot at the time about how MLPs are like decision trees, but they extrapolate.

This is genuinely a new phenomenon, and you've studied it because I've seen some of the graphs. These models actually extrapolate in ways that normal models don't. What's the behavior there?

Extrapolation is super important, also for me coming from AutoML and Bayesian optimization, if you only see parts of the space and want to extrapolate to other parts of the space. For example, with scaling laws, you only see points for small neural networks, but you want to predict how well your large neural network is going to do. You need to actually be able to generalize.

You can put it in your prior that you want to generalize. Just say, “I want to train on these data points from here, and I want to be able to predict over here.” If that's in your prior—that the train and test distributions are not IID—then you can also learn out-of-distribution predictions.

I think about 3 weeks ago, Google released something called TabFM, and I took some notes about it. The first thing is, imitation is the sincerest form of flattery. They were very much inspired by the work you've done, scaled it up around 30 times, and got significantly better performance on the TabArena benchmark. What's the story with that?

As you said, it's flattering that they're also jumping on this and basically taking the same architecture, the same types of priors, et cetera, and solving it with Google Compute to get bigger and better models. It's not surprising that this is better. I think it's maybe a little premature to go that big because it also comes with some issues.

The model is very large and, as a corollary, quite slow. It's about 15 times slower than our forward pass, even though it's just a forward pass, which is super cool. The fact that it can do this just in a forward pass and get this performance is really nice to see. But it's a very slow forward pass. It's 15 times slower than our forward pass.

We have this test-time-compute TabPFN thinking that's actually 10 times slower than our standard model, so our test-time compute is still faster than a forward pass of TabPFN and is stronger in terms of ELO scores, et cetera. But it is a forward pass, and so it shows that if we look at the scaling laws—and of course we already have scaling laws—and invest that compute, we will get better and better.

They did focus quite a bit on small data, so in particular, for the smallest datasets in TabArena, it's much better than previous methods. For tiny datasets and small datasets, it's still better. For medium datasets, it's actually breaking even. Medium is up to 100,000 data points, and that was our focus with the previous models.

In TabPFN 3, we already went to other benchmarks. We went to the TALENT benchmark, which is up to 1 million data points, and to Beyond TabArena. We haven't actually focused that much on TabArena, which is up to 100,000 data points, and this is a good reminder that we should also focus a bit more on that one.

Our current benchmarks will make sure that the next model generation that's cooking is also, again, much stronger on TabArena itself.

But we don't want to over-index on it because we do care a lot about scaling to a million, to 10 million data points, to non-IID group data, et cetera. And TabPFN being such a big model, we actually just can't even run it on Beyond Arena. It just breaks and runs out of memory for data sets that go beyond roughly 100,000. But it's very cool that they're part of this now, and I'm super excited for the field to grow.

There is the question of the moat thing. You guys actually did the research, so you understand this and can presumably evolve it in interesting directions. But one thing we haven't spoken about that is part of your moat is that a lot of principled knowledge went into designing these priors, right? We haven't really discussed how you did that yet. How did you make those priors?

Prior development is a fairly iterative process. The way we started was basically that, for TabPFN 2, we had this set of holdout data sets that we wouldn't train on, but that we would use to ask: If we make a change to the prior and train a network, how well does it do on these data sets, and where are some failure modes? We could actually see, “Ah, there's a failure mode in this data set. What are some of the structures in this data set that it might be failing because of?” For example, outliers, missing values, or uninformative features, et cetera. Then we would also put those into our prior.

Over time, we just made it ever more complex to capture more and more possible ways that the data might have come about. If your prior is limited—for example, to just linear lines—then the posterior is going to be limited based on linear regression. It's a perfectly fine model, but it's just very bad if your data is actually not linear. And so we would have very complex priors that can capture as much of the world as possible.

Let's talk about causality. This is one of the things that's spoken about a lot in this context. First of all, can you explain the difference between causation and correlation? To what extent can we expect these models to be causal, and what would that even mean?

Let me explain that with an example. If you think of a medical example where you have a disease that some patients have, and these patients are given a particular type of medicine that helps them against that disease, the stronger the disease is, the more of that medicine they get. If you have a data set where you just see how much or which medicines a patient gets, and you want to predict whether they have that disease or not, then we can basically build a perfect model that says, “Well, if they get this medicine, then they have this disease.”

You might be tempted to say, “Ha ha, let's stop giving them that medicine, and they won't have that disease anymore,” but that would be foolish, right? The causal relationship is the other way around: because they have the disease, they get the medicine, not the other way around. And so that's where causality and causation are really important to distinguish when you start with these interventions.

I just said, “Let's stop giving them that medicine.” So that means you change a variable in causal ML using the do-operator. You do this: give the patient this medicine, rather than observe that the patient is taking that medicine. And those are 2 entirely different things, because if I observe that the patient is getting this medicine, then there is some relationship—the reason that they're getting it is that they have this disease. But if I just decide that they're going to get this medicine, then I know nothing about the disease. Getting ahold of that is very tricky, but that's where causal ML comes in.

This do-calculus came from Pearl. He had this ladder of causation. Just frame that up.

The do-calculus does come from Pearl. In the Pearl school of causality, you know the graph, and if you know the graph, then you can compute all kinds of things. But in practice, often you just don't know the graph. If you're a company and you want to do pricing on some of your products, and you observe hundreds of different features of your customers, you don't know exactly which of these features cause which, and so on. And you still want to make decisions in this causal space and be aware that there might be causal relationships there.

There are 2 different problems to solve. One is to make predictions while knowing about these causalities, and that's basically where our SCMs come in: We make predictions by integrating over all the possible causal relationships. I don't know the right causal relationship, but the data is suggesting this one more, so I will weight this one more, et cetera. Then I basically integrate over the space of all the SCMs.

There's also the question, “Hey, actually tell me about the causal relationships.” That is much trickier. And there's also a third question: “Well, tell me which interventions to do.” The first one was just predictions under observations. This third one is predictions of interventional effects.

You can actually also do that with these foundation models. The nice thing is that, again, all we need to change is the training process and the meta-training. During meta-training, we can control this. Usually, we sample a causal graph and observe the variables in it. Now we would again sample this causal graph and observe the variables, but then we would take the same sample, make an intervention in it, and observe the effects of that intervention. Then we have pairs: This is observational data, and this is an interventional prediction. We can learn from hundreds of millions of graphs from our prior.

If I see these types of observations and I make this intervention, this will be the effect. And so you can meta-learn to make predictions about interventional effects from purely observational data within some limits. There are some limits of identifiability, et cetera, but you would just integrate out over all the possible worlds—all the possible structural causal models that are in line with the data.

This is also a problem with language models, and it's related to this concept of abstraction, which I'm very interested in. Linguists have distilled a very parsimonious, possibly broken kind of generative grammar, or the Minimalist Program, or something like that. We have distilled that down so it's legible to us.

Because if you think about it, causality could happen at multiple levels of resolution. It could be all the way down to the light cone. What we do is come up with these causal variables. Part of the way we distill those causal variables is because we are actors in the world, so we can actually try things and explore counterfactuals. Then I guess, over time, we just distill it down, and we can imbue these into our AI models.

This is kind of what you're saying: You can have a load of core structured causal knowledge, and then you can do this prediction with a causal trick if you can latch the causal knowledge onto the ML model. But does it go without saying at the moment that the ML model cannot acquire de novo causal knowledge without us giving it to it?

In order to know something about the effects of interventions, the model needs to observe the effects of interventions somehow. So what we usually do is randomized controlled trials, A/B tests, et cetera, where we actually do interventions. Then, based on this interventional data, you can fit models.

But often you just have a whole lot of observational data, and you don't have the interventional data. What this new line of work on causal foundation models allows you to do, to some degree, is reason about interventions without observing them at test time. But you observe them during meta-training, where you say, “Well, this is my causal model that I sampled. If in this causal model I make this intervention, then this is going to happen.” You just learn the effects of that over hundreds of millions of data sets. Then you have observed a whole lot of interventional effects, and you can actually make predictions about these.

Sometimes you have non-identifiability. It might be that A causes B or B causes A, and you don't quite know which one it is. But we don't need to say which one it is; we just need to solve the Bayesian integral over: What would this one say, and how likely is it? What would this one say, and how likely is it?

When you're actually not sure which one it is and they make completely different predictions, then in the extreme you want to predict, “I don't know,” with 0.5 probability that this is right and this is my result, and 0.5 probability that this one is right and this is my prediction.

And again, you have this multimodal distribution where you actually have zero probability that it's in the middle. It's either that or that, and I just don't know the right causal graph. But in many other cases, you can actually figure out with a higher probability that it's this causal model, and then the probability mass shifts over to the predictions that that causal model would make.

So, if I've understood you correctly, you're saying that we can have high-fidelity models that could imagine counterfactuals. They can say, “I do this intervention. I could imagine this possibility and this possibility.” The model is so good that the imagined space of possibilities is quite good. So the model could then, in principle, select and create a causal model from that.

But I guess I'm trying to understand what the bar is for a causal model. Because even if we do RCTs in the real world, it's still somewhat statistically arbitrary. We might establish a causal relationship, but is it a binary thing? Are we in the domain where these statistical causal models will be nearly as good as having causal status, but not quite, or something like that?

You can run RCTs, and typically when you run RCTs, you get much better predictions. But now, with these causal foundation models, you can, in some cases, get very close to the type of performance you get if you have RCTs. There are already papers that reduce the number of data points you need to have in your RCT in order to still get the same type of performance.

That, of course, I think will completely revolutionize medicine and A/B tests and so on if, in the future, we can get away with much shorter studies, get our medicine to the market much earlier, and still have the same confidence or even higher confidence. I think there's a whole lot of potential to do a lot of great things there, and I'm very excited about this.

But of course, we need to do this properly and theoretically grounded, and this is high-stakes. The potential gains are huge, and I'm super excited about that.

And what would that look like? If you hypothetically created the—like, if your next product was something causal—what would that look like? Would it be couched as purely observational data, or would there be some kind of data scientist who would structure it in a certain way, so that the purpose of the model would be, as TabPFN does, to essentially grab that causal structure very, very efficiently?

The benefit of this causal model would be that it would be interpretable, reliable, explainable, and that kind of thing.

We already have some papers on this. We have this Do-PFN paper, together with Bernhard Schölkopf, who is an advisor of ours and one of the leaders in causal ML worldwide. He's the most cited scientist in causal ML in the world, so he definitely knows what he's doing.

There are some other teams working on this. CausalFM comes from Layer6 and the University of Toronto. We actually had the lead author of that join us last week as well.

Wow.

We have a lot of great people on the team who work on causality. Basically, you can imagine this as getting some observational data and wanting to predict, “If I change this variable here, what will happen?” If I give this patient this medicine, what will happen? What will happen to another variable—for example, the disease status? Is that going to get better or worse?

In some cases, you can actually make predictions about this. In some cases, you'll always make a prediction, but the prediction might be just complete uncertainty: “I just don't know.” But if the data actually gives you some traces of causality—basically, if it has some traces of causality that point in some direction, that, hey, it's more likely that A causes B—then we can give more weight implicitly. Of course, this is learned by the model, but then we can give more weight to the model where A actually causes B and make better predictions.

The cool thing is that we now have follow-up work on that as well, published at the last ICML, where domain scientists, if they know that A causes B, can specify that. Then we build the posterior over all structural causal models where A actually causes B, and not B causes A, and not where they're independent. That allows us to really use the domain expertise of people.

There is also work that can probabilistically infer some causal structure. It can say, “Given this data, how likely is it that A causes B?” That allows you to do all kinds of things, like root-cause analysis, et cetera. I think that's really exciting. Our current models do not do that yet, but that is definitely something we're working on for the future.

We just started our open-source research arm, where we collaborate with anyone in the community on tough problems like this. I think these problems are too exciting and too important for humanity for us to just work on them by ourselves. We should really work together with the greatest researchers in the world in order to tackle these problems, and then, of course, we'll build products around them.

It's really important to solve these problems at the core. We don't want to have any red line in the way, but really be able to work with anyone in open source.

That's really exciting. Just one final question on that bit: what is the potential for causal structure learning? At the moment, we decide what the variables are and give them names. They're legible, parsimonious, and whatnot. But I can imagine a future where an AI would be able to come up with something that was very high-resolution but much better.

Is there a trade-off between the fidelity of the causal structure and its intelligibility and explainability to us?

What LLMs will come up with is, of course, the world knowledge and causal relationships in the world. I do not think that they will come up with proper causal relationships between features that we see from the data. If you have tens of thousands, hundreds of thousands, or millions of data points, again, it's in the numbers, in the statistics, and that's what LLMs are not strong at. I don't see them getting stronger at that anytime soon.

Just on that, that's a tangential question. LLMs are incredible. Did you read about the Erdős unit-distance conjecture? It was 125 pages of chain of thought. What they don't seem to do—and the reason why we are part of the process—is that we can look at the spaghetti and say, “That's basically this.” We can abstract it, and we're doing more with less. We're compressing. Why don't LLMs do that?

Honestly, I think that is something they will also learn to do, because LLMs do compress their knowledge. If you read a billion tokens, you can't keep that in your memory. You need to compress it somehow, and they will. They will learn to do this better and better and, I don't know, form lemmas and whatnot that they will then remember.

But that is still in the space of language and not in the space of the data and the numbers. I think the challenge will be to bring the two together and really use the world knowledge, use the amazing chain-of-thought capabilities, et cetera, but also bring in the causal information that we can see from the data.

We promised we'd come back to multimodality later. Some of the things you're looking at now are text data, relational data, graph data, and that kind of thing. How does that play out in TabPFN?

Relational data right now is an interesting one. We haven't actually worked on this much at all, but we got this for free from the open-source community, who developed an embedding of a relational database.

These are queries in a relational database where you want to predict some entities in one table, and you want to take as context all the connected tables, where you have other tables that tell you something about this particular entity. You can actually have an embedding that takes that relational database query and puts it into a single table, and then they just ran TabPFN.

In this case, I think they ran TabPFN 2 or 2.5, and we took that method, put a little bit on top, and ran TabPFN 3 with it. What we did was build better benchmarks and so on to make sure that we're optimizing the right objective.

And yes, TabPFN 3 out of the box was actually the best foundation model for these relational databases. We evaluated this on the original RelBench from Kumo. It was better than KumoRFM, which is their own relational foundation model that also has an individual tabular foundation model at the end. We also worked a bit more on making a better version of RelBench, which we're releasing next week. I think when this interview airs, it will have been out for 3 weeks.

One interesting thing is that we were talking earlier about there being a bunch of priors in the model. For example, it might have a locality prior. It's got columns and rows and whatnot. What would it mean to put in other types of geometrical priors? Could you, in principle, just put a relational prior in there, and it would understand that? You could put in multiple data tables, and it would understand the relationships between them and somehow implicitly do joins and all of that kind of stuff, or maybe work with graph data or tree data. Is that possible in principle?

Yes.

Okay.

Basically, this is clearly the name of the game: building priors. You look at the types of data you want to be good at, put that in your prior, generate your data, and then the model is going to be good at that. Of course, you need to solve some architectural challenges. If you have different relational tables, how exactly do you put this into your network? How do you make sure that this is efficient? But the sky's the limit there. I think this is totally going to happen.

How do you see this playing out in the enterprise? I think there is a bit of an issue in the enterprise in that there are security boundaries. Sometimes tabular data, and operational data in general, is the hardest to get.

Speaking personally, I can't wait to go into QuickBooks, download all of my data, and stick it into TabPFN. But in the enterprise, how do you see this playing out?

Tabular data is the most common modality in the enterprise. Nevertheless, historically, there haven't been these tabular foundation models out there. People use traditional models like XGBoost, and now they're being exposed to coding agents and LLMs. Many newcomers try LLMs on their tabular data and see them fall flat on their nose, because they haven't seen this type of data. This data hasn't been on the internet, so the models haven't been trained to work well on it.

There is a big opportunity there. That is where tabular foundation models are required. The more tabular foundation models learn about the data in the organization, the better they will get. You can also take a tabular foundation model—our models are open-weight—and fine-tune it on the data of the organization. It will get much better performance for that particular data, because then you can extract the type of intricate patterns in that type of data and work much better for it.

How do you think an organization should design structures around this? For example, there might be some kind of data platform. Do you think that individual people in the organization should be building out models like this, or should there be some kind of platform team? How would you do the data governance?

There is a wealth of tabular data in the organization. Some of it is in Office 365, and some of it is just operational data. It might be a retail system over here and a financial system over here. Essentially, what you want to do is make this data available, but there also need to be controls. How would you do that?

It's going to look pretty similar to LLMs because you don't want to make your data available to everyone. You also don't want individuals training their own LLMs. That doesn't make any sense. Rather, you want one part of the organization really understanding these models, knowing how to fine-tune them, knowing where they still fall short, where to bring in some other data, and where to check whether XGBoost is still better if you have a billion data points or whatnot.

I see this as pretty centralized, with services that are being offered. Of course, as Prior Labs, we want to support that. But we need to work with the organizations on how to actually play this out.

What is your revenue model? As I understand it, at the moment you can just use your Claude agent, and it can use TabPFN to do a bunch of stuff. What are you building as a commercial offering?

You can use TabPFN in many different ways. You can use it from the open source with a non-commercial license. You can try it out, test it, and it's entirely free to test it in a non-production setting. If you want to use it in production and make money with it, then you need to pay a license fee.

That is one business model, and then we have all kinds of things around it. We have thinking mode and scaling mode, which work better for large data. We also have models that work particularly well for very small data. Of course, we have services to get people off the ground and get them better performance with tabular foundation models. We have fine-tuning as a service— all kinds of layers on top that you can stitch together. Ultimately, you can think of it as a platform that just makes a data scientist dramatically more efficient.

I'm imagining a couple of years ahead. You might have a forward engineering team. You might eventually have models that are too big for customers to serve themselves, so presumably you might offer some kind of serving stack for that. The fine-tuning thing is interesting. We didn't talk about that, but do I understand that you can fine-tune the models, but customers can't yet fine-tune their own models?

The models aren't too large for customers to serve yet, but we might be able to serve them faster. We might have different tricks. We already offer an API where we control the infrastructure, and we can make sure that we batch different predictions. We can make sure that the GPUs are kept busy. If a client were to reserve a GPU in order to do all the predictions, that GPU might idle all the time, and it might actually be much more cost-efficient for us to do the serving than for the client to do it.

This API is also part of the commercialization. We have private VPCs for data that cannot leave the organization. But if you already have a cloud partner, then it can run there. For example, you can already use it with Azure and AWS. And then, of course, in SAP's Generative AI Hub, we have TabPFN. So there is a variety of different ways to commercialize.

We're particularly good at fine-tuning. There are also some fine-tuning wrappers, so people can fine-tune the models. But again, if they fine-tune the models on the data in their organization, there is already a license for the model, and that also applies when they fine-tune the model. Fine-tuning is something we can do, or the customer can do. Both would work.

What would happen if OpenAI or Anthropic just embedded it in their harness? Would you charge them straight away, or would it be based on how it was used by the customer?

That's going to be interesting to see how the token economy works out. I think tabular prediction tokens will be much cheaper than the LLM trying to do it themselves. You can have an agentic agent that goes and writes some code and gets something that may be close to that, but that takes a lot of time and tokens. Rather, you could just call a tabular foundation model, and it should do it faster and better and for fewer tokens—or for an amount of tokens that costs you less.

In this agentic economy of the future, that's what we're developing the methods for. They should be better than any other method. LLMs call calculators because they're better at doing math. Just like that, they should call tabular foundation models because they're better and cheaper at doing this than they can do it themselves.

I'm just imagining that if someone from Anthropic were listening to this, their first thought would be, “I'm going to update the system prompt and just use TabPFN if anyone's using tabular data.” Or if most normal people at home listen to this, they're going to be thinking, “I'm going to update my Claude global skill to use TabPFN anytime I'm working with tabular data.”

Yes, absolutely do that.

We have skills files and everything.

Yeah. I mean, it just seems like a no-brainer. The other thing is, you're launching a research arm. Tell us about that. What are you guys doing?

Yeah, so we're launching an open research arm. We're already doing a lot of research in Prior Labs, and we want to do a lot more fundamental research also because we just got acquired by SAP, and we have a lot of funds available to do a lot of cool stuff.

The leading motive for the acquisition has been DeepMind, and so we want to be the best place in the world to work. We want to publish up a storm. We want to be the place where everybody goes in order to do the greatest work, and good things will happen from there.

In the open research arm, it's going to be fully open research. We can collaborate with anyone in the world, with universities, with the ELLIS Institute, and with other partners. I also still have a university group at the University of Freiburg. We already collaborate with Layer 6, for example, in Toronto, the University of Toronto, with Singapore, and with the ELLIS Institute in Tübingen, et cetera, et cetera. All of those should just be possible super freely and easily.

Basically, we're going to take what we do at the university and scale that up, and also do that in Prior Labs. It's going to be, of course, super synergetic for people if they want to have impact and then also see some of their work being moved into a product.

We also want to tackle moonshots, so really take high-risk, high-gain problems that, if we solve them, could really move the needle for humanity. If you're listening and you have data for that—if you could do tabular predictions better for this data, maybe super-complex data, and you could, I don't know, with this, then help cure cancer or help global food production or help with any type of problems that are good for humanity, please get in touch. This is definitely something we want to really focus on and do some cool stuff.

Amazing. To be clear, you are hiring researchers and interns. I'm excited about this because we hear too much about America and China dominating the AI space. We're here in the south of Germany, in Freiburg, doing frontier AI research. There are loads of talented folks in Europe. Do they give you an email, Frank?

Please go to our website and apply. We have over 11,000 applications—

Okay, okay.

—and have hired 45 of them. If you're excited to work in that type of team, where we have the luxury of a whole lot of people already applying and are able to pick the world's best folks in the space, then by all means, please apply at priorlabs.ai/careers.

We hire for pretty much every role you can think of, in particular now also research scientists who do model development and forward-looking model development. You can think six or nine months out and build better architectures, et cetera, but also work in the completely open-source arm, where you can collaborate with anyone in the world.

Amazing. Frank, thank you so much for joining us today. It's been great.

Thank you so much for having me.

As many of you may have seen, since we recorded this interview, Prior Labs has now released a new version of TabPFN, version 3.5, and it's looking really, really good. I got in touch with Frank, and he has recorded a very short update for us, which I'm going to tag onto the video now.

We just released our newest model, TabPFN 3.5, and with that, we evaluate on much more than TabArena. I'm super happy to say that we took first place on 7 different tabular-related benchmarks, including tables with text, multimodal data, relational data, et cetera, covering a lot of the problems that data scientists actually face.

We didn't focus on TabArena that much, but also improved there. TabPFN 3.5 Pareto dominates other tabular foundation models—for example, being 20 times faster in a forward pass than TabFM for the same quality, or being over 100 Elo points higher while still being faster.

More importantly, we improved dramatically on the Beyond TabArena benchmark that I mentioned, which is much more similar to the types of data complexities that data scientists actually face every day, with group data, temporal data, data scales from tiny to 1 million data points, text in the tables, high-cardinality features, and so on.

As a result, TabPFN is now also much more relevant to data science competitions. For example, our research scientist Nick Erickson, who previously built AutoGluon at AWS, tried TabPFN 3.5 on the historical AutoML Challenge from Kaggle from 2015, which had 3,500 competitors and a prize pool of $10,000. It was won by this multilayer stacked ensemble of 36 different models with handcrafted feature engineering.

Nick had used this competition to improve AutoGluon over the years, and with the AutoGluon paper, he had actually gotten a top 1% solution in 24 hours on 96 CPUs. Now he used TabPFN 3.5 and got the number-one-ranked solution in 1 line of code, in 1 minute of compute, on 1 RTX PRO 6000 C GPU.

I'm super excited about TabPFN 3.5 as a practical tool for data scientists now.

So there you go. That was the update from Frank on TabPFN 3.5. I think it's a genuinely exciting time to be alive because you can just take this foundation model and apply it to so many real-world problems. I've even applied it to the spam filter on our Discord server, which is absolutely amazing.

Admittedly, you need a fairly beefy machine to use it, and you need quite a lot of memory on the machine. But it really, really is cool. I hope you folks have a play around with it. I think it's free for non-commercial use as well.