The Dawn of Dynamic AI: RFT Comes Online, w/ Predibase CEO Dev Rishi, from Inference by Turing Post

Cassia

Hello, Dev. Thank you so much for joining me today.

Of course. Happy to be here, and thanks for having me.

Cassia

Let's start with the big picture. When will we train once and learn forever?

It's a great question. I think that world is actually here today. Most of the time, when we see customers using models in production, they're taking a model on which someone else has done 99% of the heavy lifting, and then they're doing a last-mile 1% customization.

The trend that's going to move toward what you said—train once and then learn forever—is a shift where people stop using a static model, this one model someone else trained, and instead have a pipeline that allows them to improve the model continuously while it's in production.

We've started to see some of our early customers already put these types of pipelines into practice, and that's what I'm most excited to build toward as well.

Cassia

That's super cool. One of the things that you initiated at Predibase was the post-training technique RFT. Can you tell us a little more about it? Do you think it's the big unlock, or is it just another tuning trick?

It's a great question. I think we were the first end-to-end platform to offer reinforcement fine-tuning, or RFT, when we released it a couple of months ago.

Cassia

Sorry for interrupting. Can you also briefly explain what it is?

Reinforcement fine-tuning takes a different approach toward the way that you can fine-tune or customize models. The underlying intuition is that, rather than needing large amounts of labeled data, which is what you need for traditional supervised fine-tuning, you can actually fine-tune with really small quantities of data. Think about a dozen examples or so. Instead of labeled data, you add this concept of reward functions.

Reward functions are essentially rubrics that an individual customer can write to help grade a model's output. The idea is that the model will learn how to adapt its behavior toward the types of things that you want to incentivize or reward.

As an example, if you're teaching a model how to write code, you might write a reward function that says you'll get 5 points if you get the formatting correct, another 10 points if it compiles, and another 20 points if the unit tests pass.

In this way, you teach a model how to generate its outputs based on objective criteria. The goal with reinforcement fine-tuning is: if you can measure it, you can improve it.

You asked whether it's a large shift that we see for the future or just another tool in the toolbox. Honestly, today reinforcement fine-tuning is one of a few different techniques that's really helpful for customers who want to start tuning their models. But I think where we're going with reinforcement fine-tuning is going to fundamentally shift the way that people customize models.

Today, reinforcement fine-tuning is a one-off training process. But where I see RFT really going is toward becoming part of this continuous feedback loop, where models are getting better online. I think that is going to be a paradigm shift in how customers tune and train models.

Cassia

Do you already see companies implementing this feedback loop?

At the very early, cutting-edge end, yes. I'm working with a couple of companies in the healthcare domain that are building copilots and assistants for their patients. They have a lot of interaction data with their patients, and they're bringing in a combination of LLM-as-judge systems to verify how good those conversations were, along with clinicians who are able to label different conversations and bring that in as continuous feedback.

Rather than needing to spend months doing the labeling, they're starting to take the labeling that can be done over a handful of conversations and feed that into an RFT loop. Today, it's very early, and I only see the most cutting-edge companies really doing it. But this is the type of pipeline that I think more and more companies are going to use as we move toward more continuously improving models.

Cassia

What's in the way of them starting to do it?

It's a good question. I think the biggest thing is a feature that we're working on. There are actually 2 things.

One of them is a feature that we just launched in Predibase: a very simple way to collect prompts and responses from any of your production deployments automatically. One of the key things with tuning and training models always tends to be data, so the very first feature we have makes it easier to construct these data sets using live production traffic.

The second thing is making it easier to learn from feedback. Rather than needing large amounts of labeled data, how do you directly nudge a model with small amounts of feedback? There are lots of techniques in the platform, like DPO, or direct preference optimization, and others that help you learn from feedback. We're also working on some novel techniques on the research side at Predibase for using reinforcement fine-tuning with user-feedback data.

Cassia

Is anything coming soon?

I think you can expect us to publish a little bit more about this in the next few months.

And really, what we're going to be talking about is what we've experienced with real-life agentic applications in particular, and the type of feedback that users are looking to be able to give. How do you systematize that into a continuous stream? The very first thing we'll release is just going to be showing how small amounts of feedback can make performance impacts, and then you can scale out those performance impacts with more and more feedback you get.

Cassia

What's your perspective on agentic workflows and agents in general?

My perspective on agent workflows is that they're in the very early innings, and so a lot of things that people are building are a little bit brittle today. The very first thing we need to do when we talk about agentic workflows is really define what an agent or agentic workflow is.

I think about an agentic workflow as having 2 key components. The first is having a chain of multiple LLM calls. If you think about a single—let's imagine you're doing document classification—we don't think about that as an agentic workflow because it's a one-shot process. Whereas, if you're having a conversation with a bot, for example, to understand your medical diagnosis a little bit better and schedule a follow-up, that involves multiple turns. So the first piece, I think, is multiple turns, multiple calls.

And then the second piece of an agentic workflow is that it will likely have the agent be able to do tool calling, make calls to other functions that it can actually use to fulfill a request on behalf of the user. My view is that right now, the way that these agents get built is quite brittle. A lot of times, people have built what I think is a really compelling demo that works well if you are on the golden path.

But if you go off the golden path, then the model is triggering a number of different failure modes. It's a very simple thing: if you're only 90% accurate on a given LLM call and your LLM has to make 5 different calls, you're already sub-50% in terms of the user experience. So that last mile of quality with agentic applications becomes really, really important.

I saw this when I was a product manager on the Google Assistant back in the day, which was one of the first AI agents—not using generative AI, but using more classical NLP methods. The bridge we need to make as an industry is getting to more robust agentic workloads.

Cassia

Thank you. Yeah. Speaking about Google and your previous experience, you're a product-first founder in a very research-heavy space. Is it hard for you? How do you keep up, and what areas of research are you following most closely?

I really enjoyed it, actually. My background was—I did my undergrad and master's in computer science, and had done some initial research as well in CS before, as I like to say, I sold out and became a product manager.

My real interest in product was because I saw it have kind of the most cross-functional impact. In research, you usually go very deep into an individual space, whereas in product you get to see how a combination of research works with engineering and design. And so that was my main intent.

Even when I was at Google, though, I was working closely with research teams. And so I've always felt very comfortable working in areas that are fundamentally based on things that are still developing and core research that's happening.

I will say this space moves much faster than anything else that we've seen—much faster than, I think, when we were talking about some of the shifts to cloud or shifts to mobile. The pace here is truly that you will have a breakthrough in expectations about every week.

When we were planning our reinforcement fine-tuning launch, one of the most stressful things was knowing, is OpenAI or Anthropic or Mistral or DeepSeek or Google, Amazon, or Meta going to put out something massive in the same week and just suck all the oxygen out of the room? So it is dizzying to be able to keep up, but I think some of the core background and research that I had helps a lot for just being able to understand some of the core techniques people are employing here.

Cassia

I remember working with a few ML companies before the ChatGPT moment, and when it happened, so many needed to pivot. If we go back to November or December of 2022—

2022 was when it came out.

Right, right, right. It's just so fast; it seems like a long time ago. What changes did you have to make when everything started to just explode?

It was a really interesting moment for us because when we started the company, we had a mission to democratize deep learning. We had built interfaces and a product and infrastructure to make it easy for people to train their own deep learning models.

The way I like to reflect on it is that, at the end of 2022, OpenAI democratized deep learning more than any of us. It had done it through these large, pretrained deep learning models that were able to start having conversations in one shot.

The reason I say it was a really interesting position for us was that, as a deep-learning-oriented company, we had started to see these types of workflows already be popular on our platform, but at a much smaller scale. In early 2022, the most popular piece of functionality on our platform was that you could pick a pretrained deep learning model, like BERT as an example, and you could fine-tune that model on your data.

The reason that people were coming to us was already because they wanted to start to adopt some of these pretrained transformers and adapt them toward their data. But the types of use cases, the user journey, and the persona just fundamentally shifted.

Cassia

Right?

When we were talking in 2021 and 2022, it had to be an NLP engineer—someone who understood the intricacies of a BERT or a T5, or an image model in computer vision like ViT—in order to be able to really understand the platform. Fast-forward to today, I think some of the most prolific AI engineers are ones that just got started in the field a year or 2 ago.

We really needed to—depending on how you think about it, I often think about it as a complete pivot—but it was a real focusing in on our product to say, rather than, "We're going to help you build deep learning models across the world in general," we decided to really pick and make the bet at the beginning of 2023 on just this one technology: large language models.

We decided to make the bet that the future of LLMs in production was going to be specialized and customized. And so we wanted to build that tuning and post-training stack to make that happen. Then what we quickly found out was inference was going to be a huge part of this game, and so we went really big in that world as well later on in the year.

Cassia

There's so many things to unfold here, but let's start with more narrow AI. Do you think the future for companies is smaller models that they'll use for their specific cases? How do you think about it?

Yeah, I think that genuinely the entire pie of AI use cases is going to grow.

So if you ask me today versus in 2027, you're going to see more use cases of every breed. The difference that I think you're going to see is that my favorite customer quote is, “General intelligence is great, but I don't need my point-of-sale system to recite French poetry.”

Mhm.

It's this idea that most customer use cases look like something one of our customers, Checkr, does. They look through employee background checks and are looking to extract very specific information with respect to criminal codes and violations, and previous things in employee backgrounds. It's not like they need the model to be able to do French poetry and write Python code. They need to do a really high-quality job on one particular set of tasks.

So in enterprise, I think the majority of use cases are going to move, and have already started to shift, toward a lot of these narrow use cases that are automation-oriented. Not to say that these general-purpose agents where you can talk to them about anything in the company won't exist and won't be great demos, but some of the high-value use cases are going to look like being a really prescriptive, specialized agent that understands how to solve a series of tasks very well.

And I think in enterprise, that's going to be true. In consumer, I think it's a little bit harder to say, actually. I think versatility is actually very helpful when it comes to consumer. But regardless, one thing we've seen that wasn't obvious in 2023, but I think is obvious in 2025, is that we aren't going to live in a world where one model rules it all.

You're going to see a mix of open-source models, closed and commercial models, different parameter-size ranges, and, just like any software tool, people are going to choose the best tool for their individual task. The thesis we have is that the types of tasks that require narrow AI are going to grow at an even faster rate than the types of tasks that the rest of the AI world is dominating with today.

Cassia

In this crazy race, do you have a plan? How far in the future is it—2026, 2027? What's your strategy and planning?

Yeah, it's a good question. I think there's the famous quote from boxer Mike Tyson: everyone has a plan until they get punched in the face. That's probably true in AI as well. We absolutely have a plan that we think about extending through the end of this year and through 2026, but the truth is AI changes on a weekly basis. And so you need to be able to have a framework that allows you to make decisions as things come in more quickly.

It's all guided by, I would say, a north-star vision. Our vision is to help customers develop specialized AI and help them tune, serve, and deploy their models. The pieces that are more dynamic are what the best ways will be for people to tune their models. We're not religious about saying supervised fine-tuning is the be-all and end-all. We just think it's the best technique today if you care about getting the best performance out of your models.

But in a year from now, you could see a completely new technique come up. Obviously, in 2025, the biggest new technique is reinforcement fine-tuning, which we pioneered at the beginning of this year and had talked about earlier, too. And so I think from our standpoint, our vision is really going to be predicated toward the goal of helping customers develop specialized AI, tune those models, and then run highly performant model deployments in production. So that means we're going to continue to build infrastructure in training, inference, and serving.

And we know the things that are going to come out through the rest of this year: advanced techniques for how we specialize models and customize them, advanced techniques for how we run inference better, and expansion to modalities. We're seeing more customers ask us about things like multimodal, vision, or voice today. And so we're going to want to continue to expand that within this broader framework of tuning and serving. We'll see where the research really leads and be able to adapt that back into the platform.

Cassia

Let's go to inference. Why is it so hard for enterprises, and what could make it easier?

Yeah, inference is a phenomenal example of something that starts off being very easy and then gets hard as you actually peel back the layers of the onion. Why is inference hard? It gets hard between the different stages that an organization ends up being in. There's the crawl stage, where I'd say the difficult part of inference isn't so much needing brilliant software engineering, but it is that GPUs, for example, might be hard to get.

So if you have the largest possible model, you might need 8 or 16 H100s running to just have a single replica of a model deployed, which means you're going to have to procure them and then decide if you're able to autoscale them up or down inside of your environment. And you need to set up an initial inference server and framework.

A number of companies, including us, have tried to make that easy by open-sourcing inference frameworks and servers. So we've open-sourced LoRAX, which is the underlying inference technology that we use, so that anyone can stand it up themselves. I think what I see, though, is if you're a smart engineer, you can set up your own inference framework and server. You can get that working, and it'll work well to start to feed your initial prototype and application.

What's really hard, though, and what I think a lot of customers don't want to do, is maintain production inference. And production inference is another ball game. That doesn't mean that the model works and is up 95% of the time or even 99% of the time. It means it's backed by 99.9% or 99.999% SLAs. That means you need to have resilient fault tolerance and be able to do blue-green deployment updates. That means you need to be able to do multi-region replication in case you have these model deployments set up.

And then, most critically, GPUs are expensive, which means you need to be able to optimize the models such that you're actually getting the most for every individual token that comes out. And so you're optimizing for total cost of ownership, whether that's using a small model or using some of the techniques we have in our platform, like TurboLoRA, which are just software-defined ways to increase your model throughput by 2×.

All of these factors come in when you make that shift from, “I'm prototyping and able to get inference going,” versus, “I'm actually going to production. I need monitoring, SLAs, high-performance throughput, and all the other functions that go with the fact that this is feeding a business-critical application.”

The last thing I'll just briefly say with inference is I think that, while all those are challenges, there's a view that inference is going to get increasingly commoditized as a market. And I don't disagree with this on base-model inference in particular. There's no reason that one company's DeepSeek endpoint is way better than another company's DeepSeek endpoint.

What I think is going to be really interesting is the trend toward what we kind of call internally intelligent inference. Intelligent inference, in my view, is back to that initial conversation we were having about whether you have an inference pipeline that hooks into a post-training stack that lets your models get better over time.

That's really the future of where we see inference going.

Cassia

It's super interesting—intelligent inference. We'll work on the branding and marketing for that one. But it's definitely the biggest trend that we see get unlocked by having a single place where you can do post-training and inference.

Cassia

If we speak about open source, and you are such a proponent of open-source AI models, what is missing in the open-source AI stack, and what are the gaps between this model zoo and production?

I think that the open-source model stack has gotten pretty good for some of the core infrastructure that you need to set up. I think open-source fine-tuning frameworks like ours—Ludwig, as an example—are pretty good at helping people start to run experiments. Open-source inference frameworks like ours, like LoRAX, are pretty good at helping people do some of the initial serving.

And then when you want to make the shift toward a managed platform, you have a pretty easy on-ramp to a platform like Predibase that gives you the batteries-included GPUs and infrastructure out of the box. I think one of the things that is missing is a really resilient way to do evaluations.

This is something that has been talked about quite a bit, I think, in the LLM industry. And the truth is, it's a challenging problem because LLM outputs at times can be objective and at times can be quite subjective. What's to say if something is a good summary versus if you got decent classification accuracy, as in traditional machine learning?

So I've seen a number of friends and a number of folks who have started frameworks or even companies in LLM evaluation, but I still think it's an open problem. A lot of companies build very in-house evaluation systems, and “The Leaderboard Illusion” showed us that crowdsourced evaluation cannot really work at this moment, right?

Yeah, I see people tackle evaluation in a number of different ways. Most people I see do evaluation in-house. I'll just start off by saying that the most common thing that I see—I've seen 3 versions of evaluations. The first is that they rely heavily on existing data and some proxies.

A good example is if you're doing document classification, you look at some historical holdout data and you're able to see how that model performed. That's the cleanest, simplest way, but it isn't always possible because that works for use cases where you have that historical data.

The second way I've seen evaluations done is to leverage GenAI itself more heavily. Here, we see people use LLMs as a judge as the most common technique. They start to use larger models as graders to understand: Did this response answer the customer's question? Was this a good summary for the output that we were looking for, and so forth?

Finally, I would say that the third way we see evaluation done sort of feels like vibes in some way. You'll ship the product and try to collect some sort of product feedback to help you understand whether or not the model was doing the directional type of behavior that you were anticipating. I think the truth is that a lot of evaluation is in-house, and we should not underestimate how much of it is vibes today. But getting good evaluation is going to be critical toward building this kind of continuous improvement loop.

Cassia

I agree. But it's so hard, especially with the closed models, because they publish a new model and it's a new persona. You need to develop basically a new vibe to understand it. It's really tricky.

Yeah, I think open source is helping a lot, in particular with open-source reasoning models. One of the big things was exposing the reasoning tokens themselves. That happens if you use DeepSeek R1 versus the earlier generation of reasoning models from a closed provider. You can actually start to run evaluations not just on model output, but on what series of steps it was taking in order to get there.

Look, I think the point was actually much less obvious in 2023, but it has become really obvious for most companies now: open source is here and going to be here to stay. In 2023, when we talked about the future being open source, the best open-source model at the time was GPT-J, and it was a far gap from where GPT-3.5 was at the time.

Today, DeepSeek R1 or V3, Qwen 3, and Llama 4—these models are not only on par, but many times, actually, in the benchmarks, doing even better than the leading commercial models. To me, that's 6 months ahead of schedule. I would have thought the end of 2025 would have been the earliest that we'd see open-source models beat commercial models. I thought they would be on par this year, but it's incredible, the rate of innovation that we've seen in open source.

Cassia

You never speak about AGI. What is your stand on that?

I think AGI is something that's far from the world that I see, in honesty. The world that I see tends to be, rather than artificial general intelligence, practical specialized intelligence. I think that AGI is often thought about in the research labs, where folks will come up with good definitions. If we take a definition of AGI as, can a model pass a Turing test, I would suggest that we're probably in that ballpark already. But what's the practical implication of that?

I don't spend too much of my time thinking about the Terminator-style scenarios, but I do spend a lot of my time thinking about what having this generalized intelligence looks like when you actually have business processes, like a Fortune 200 company such as Marsh McLennan, or another organization that I mentioned earlier, like Checkr, or any of these other companies that have a lot of productivity they've unlocked through business practices over the past decades. That productivity is about to take a step-function change and increase.

To me, that's really the interesting area for where this is going to go. I think there are probably some deeper philosophical questions about what will happen over a 5-, 10-, or 20-year period. We've found it pretty hard to even predict what's going to happen 18 months from now in AI. That's really where a lot of my focus has been: what's going to be the practical implication for both enterprise and consumers?

Cassia

That's a very nice perspective. What concerns you and excites you the most about the future that you're building with Predibase?

I think what concerns me probably comes back initially to the evaluation piece. People see an incredible amount of value. I see statements from analyst reports and others from time to time that often ask, “Is generative AI a bubble? Are people actually seeing real business value?”

I never get that question from a CIO at a Fortune 200 that's actually working on generative applications. When you're on the ground and actually see what these models can do, I think the question of whether there's going to be ROI versus the 50 cents or dollar that a million tokens are going to cost to process isn't even a question in the vast majority of situations. If it is, it's really just a question of selecting the right use case.

What concerns me is that we might go through a little bit of a hype bubble where people are really excited about these far-fetched, French-poetry-style use cases, where the model is doing these amazing, multi-agent-step demos. Then you enter into a little bit of a crash of disillusionment, where people have gravitated toward and attached themselves to use cases that were probably not the high-value, high-business-impact use cases that the models can do today.

The high-level thing that concerns me is if people end up essentially going too far and shooting too far ahead, without realizing the business impact that they can have with LLM use cases today. Then they enter a little bit of disillusionment out of that.

I will say, though, that given how quickly people have been able to iterate, I think I'm a lot less concerned about that than I was 5 or 10 years ago. I've been in the AI space for over a decade, and I think AI has probably been a place that, outside of the top 1% of organizations, overpromised and underdelivered between 2012 and 2022.

It was an area where we talked about how you were going to have, for example, YouTube-style recommendation systems that you could bring to your small and medium-sized business. That never really translated. I don't think that's true for the current wave of AI that we're in right now. The thing that would concern me is if we started to adopt paradigms that make people think that way.

The trend that I'm most excited about is that it used to be the way in software development that you'd perfect and then ship. You'd build, test, test, and test, do some dogfooding, and then ship out your product. As a startup entrepreneur, I think a lot about how you get really fast feedback from the market as quickly as possible.

One of my previous mentors said, “If you ship something that you're not at least a little embarrassed by, you've waited too long.” The thing that I think I'm excited about in AI is that you've actually started to see a shift in how people develop, too, where they're putting out honestly 60% solutions today. The reason they're putting out 60% solutions is that they want to test whether they have product-market fit with that solution and also start to collect data so that they can improve those models over time.

I'm excited to see what this more startup way of thinking is going to mean now that it's being adopted not just by 20- to 50-person organizations, but also by some of the larger companies adopting GenAI.

Cassia

Thank you. That's very insightful. My last question is a complete change of gears. What is a book or idea that shaped your thinking? It can be related to machine learning or completely unrelated.

Completely unrelated.

I would say a book that I like is called The Happiness Advantage by Shawn Achor. He’s a psychologist at Harvard who studied positive psychology in the context of organizations. What he found was that it wasn’t necessarily that success brought happiness in all cases; it was that happiness actually made you much more likely to be successful.

The book really covered 2 key things. The first was how having a more positive and happy outlook allows you to do better in terms of the different tasks that you’re trying to do, whether it’s work or personal. The second was ways that you can—I don’t want to say “hack happiness”; that sounds very San Francisco biohacking—but ways that you can essentially put yourself in a position to be a lot happier without having to rely on these external factors.

I think The Happiness Advantage is definitely one that I really enjoyed as an overall read, and it’s an idea that also extends to both personal and professional work.

Cassia

So you think we’re a pretty happy organization?

I hope so. I mean, I think that the truth is, if you’re working in generative AI today, it’s a noisy environment. It’s fast-moving and competitive. Players, including us, are well-funded, which means that you have a lot of cards on the table.

But I think that we and other organizations will also do our best work if we’re excited about the future that we’re running into, rather than if we’re operating out of, for example, predominant concern or other things like that.

Cassia

Great. Thank you so much. That was wonderful.

Of course. Yeah, I really enjoyed the conversation today, and thanks again for having us on.